Just in case, the previous discussion is at #6815

One question that might be relevant for this PR:
I saw that the image-editor of Delta Chat for Android has 2 resolution-limits, and those are either higher than one or both media-quality settings in Chatmail Core.
Do those apply to the image given to Chatmail Core, or is it only for display in the editor?

If those apply to the image, that could result in downscaling twice; first by the image-editor, to 1536x1536, and then by Core, to 1280x1280, and it would also mean that the change in this PR would apply to all images edited in the Android-app, because the upper limit is 1536x1536.
If that is the case, it could be a good idea to make the limits identical, so that downscaling is only necessary once; which would improve quality and reduce processing-time. But if those should not be changed, the limit in this PR should probably be reduced to 1400x1400 or less, because otherwise this change would effectively be identical to always using bilinear-resampling for the Android-app of Delta Chat.

I set this PR back to draft, because #7822 can change what is appropriate to do in this PR. With those changes, images with a resolution of up to 1536×1066¹ would not be resampled, because that is a lower resolution than 1280x1280².

I think, that the "4+ times slower" performance-test-result may be outdated, or perhaps was measured incorrectly.
I recently compared the performance of these resampling-functions (though i did not use Chatmail Core for that), and while the thumbnail-function was twice as fast as the resize-function with triangle-resampling when using a debug-build, it was only about 10% faster when using a release-build.
triangle-resampling with a debug-build was 4 times as slow as with a release-build though; which matches the mentioned time-difference.
Can someone with a sufficiently slow computer test the performance of simply always using the resize-function with triangle-resampling, while making sure that Chatmail Core and the "Image"-library have been built as a release-build; and also mention the used CPU, so that it is known what the result is for?

Can someone with a sufficiently slow computer test the performance of simply always using the resize-function with triangle-resampling, while making sure that Chatmail Core and the "Image"-library have been built as a release-build; and also mention the used CPU, so that it is known what the result is for?

Not sure my computer is slow, but i'm going to recheck this. Maybe you can also try QEMU.

See #6815 , the reason why it was closed wasn't only performance, but also reduction of sharpness when using Triangle which is a bilinear interpolation AFAIU. Quality in image resizing isn't a "linear function" unfortunately, so when we improve one aspect, we worsen another. It's good that Triangle is a fair algo in relation to source pixels, OTOH the resulting image is more smoothy (because every source pixel now contributes to multiple destination ones; this is an unsolvable problem overall), images are even get smaller in byte size, so there's definitely a significant loss of information. To compensate this i propose to increase {BALANCED,WORSE}_IMAGE_SIZE limits at least. How much -- measurements are needed.

EDIT: @72374 , nothing has changed for me since #6815 (comment), have resize() ~3.06 times slower.

... The code comment says it's 4+ times slower, obviously it depends on hardware and i'm not sure the difference is bigger on old hardware, probably on new one you just won't notice the difference.

... Also i see more blur and less details in real images. An improvement i can notice only on synthetic ones which trigger aliasing effects on purpose. I don't want to say that Triangle is worse, but it definitely needs an adjustment of the dimension limits.

... One can use this for testing:

                    let now = std::time::Instant::now();
                    let new_img = if crate::tools::time() % 60 < 30 {
                        info!(context, "thumbnail()...");
                        img.thumbnail(target_wh, target_wh)
                    } else {
                        info!(context, "resize()...");
                        img.resize(target_wh, target_wh, image::imageops::FilterType::Triangle)
                    };
                    info!(context, "time: {}", (std::time::Instant::now() - now).as_nanos());

Thank you; for checking, and the code to measure the time. :)

I used the code, to measure the difference for sending an image in JPG-format with a resolution of 12.5 MP:
The CPU that i used, is a "Core i7-12700K" (a rather fast desktop-CPU), while using the energy-saving performance-profile, which made it about 10% slower.¹
The difference, for only the resampling-functions, was consistently about 330% (~462 ms / ~140 ms), but the difference between the beginning and end² of the check_or_recode_to_size-function, was only ~40% (~1083 ms / ~772 ms).
When using only a single efficiency-CPU-core, the difference for the entire function was ~48% (~1339 ms / ~902 ms).
So, the practical difference was 40-50% (0.31-0.43 s), for an image with a very high resolution.

Regarding the proposal to increase the resolution-limit:
I would appreciate it, if the resolution-limit for BALANCED would be set to 1440.
1440 * 1440 is the same number of pixels as in the "FullHD"-resolution (1920x1080), which is a very common screen-resolution; and images with that resolution can look reasonably sharp without artificial sharpening, when displayed at typical notebook-PC-screen-like sizes (~13-18" | 122-169 ppi), or on larger screens that are viewed from a further distance. The "FullHD"-resolution is especially common for televison-screens, so it would be ideal for showing the images on those (which is relatively easy with wireless screen-mirroring).
My primary use-case, when sending images, is sharing anime-style images (often over 8 MP) and images of videogames (usually 3840x2160/UltraHD) with friends and family; 1920x1080 would fit nicely for the game-images, as it is exactly a quarter of the original resolution.

smaller in byte size, so there's definitely a significant loss of information.

The file-size-difference is probably both due to the averaging of the triangle-function, and the aliasing caused by the thumbnail-function (which is mostly in a grid-pattern, that becomes finer as the resolution-difference to the original becomes larger); though it does not mean that the image with the larger file-size contains more correct data, just that it is differently well compressable with the current encoder³.
Similarly, applying Unsharp Mask (a sharpening-effect) to an image, can also increase the file-size when it is encoded in JPG-format, even though the resulting image contains less correct data when compared to an encoding of the original.

An improvement i can notice only on synthetic ones which trigger aliasing effects on purpose.

See the comparison that i added to the pull-request-description. It is simply a screenshot of text and UI in a text-editor, cropped to fit within 2.56 MP, so the currently set condition to use triangle-resampling applies.
Neither the typed text nor the UI look reasonably good when resampled with the thumbnail-function.

The comparison in #6815 (comment) also shows some other visual issues (wavy lines instead of straight, angled lines; and rough lines that should be smooth).

See the comparison that i added to the pull-request-description. It is simply a screenshot of text and UI in a text-editor, cropped to fit within 2.56 MP, so the currently set condition to use triangle-resampling applies.
Neither the typed text nor the UI look reasonably good when resampled with the thumbnail-function.

Agree, switching to triangle may make sense. However, it's not like the thumbnail output is completely unreadable, i still see all numbers. Ok, not only "synthetic" images are improved, but also text. Btw, as for screenshots, see #7992 -- we don't want to scale down huge non-JPEGs unconditionally and screenshots are usually non-JPEGs. However, if it's an image+text screenshot, it may require scaling down because the image may be heavy.

Maybe a compromise also -- not to loose sharpness of real photos -- is to increase the limit not to 1440, but even more and a bit decrease the JPEG quality, say, to 65. However, this way the amount of JPEG metadata will grow, so we should be careful here. We can even use different JPEG quality for resized and not resized images.

So, this is a complex topic requiring many experiments, we can't change the used algo only looking at a couple of test images which we have here. Overall, i think we should maintain the same image size as before, at least not to cause new complains from users such as loss of quality or increased traffic. If we want to optimize quality/traffic tradeoff, that should be a separate change.

Here another comparison of images of text; this one in a language with more detailed characters; 日本語 - Japanese:

Image-comparison

Original:

Thumbnail:

Triangle:

Chatmail Core 2.52 - Thumbnail:

With the thumbnail-function, the "本"-kanji in the third row are changed to "木"; the bottom line of "日" is invisible in the second row after the indented rows; and "語" is changed into various different shapes depending on the position.
With triangle-resampling, the quality could also be better, but it is still good enough to recognise the kanji visually; and those can be quickly recognised again, because the shape is consistent.

Such issues with text, were also noticeable on a photo, that i compared for this.

Regarding sharpness-reduction:

More sharpness is not the same as better quality; for photos and anime-style art (typically based on line-art) in particular, artificial sharpening tends to result in less accurate depth-perception (for example: making things appear closer to the front than those should be, sometimes making things look like flat layers of paper, and/or visually detaching things from the place where those are supposed to be attached to).
Uneven sharpness caused by aliasing also does tend to attract attention, making it difficult to intentionally focus on other parts of an image.
On the photos i compared, the sharpness added by the thumbnail-function was just aliasing, that caused some of these quality-flaws; it was similar for anime-style art.

Though, #7822 increased the maximum pixel-count for images with an aspect-ratio of 4:3¹ by 33%, by 78% for 16:9, and more, if the image is less square; which means, that after a switch to triangle-resampling (without further changes), many images will still be much clearer than those encoded with Chatmail Core 2.52, anyway.

Regarding increasing the limit above 1440:

To me, that does not seem appropriately "balanced", because screens with a resolution higher than 1920 * 1080 are less common², so the improvement would often only be relevant when zooming in. I think that the current limit is acceptable (though not good) for practical use; but if an increase is wanted, it would be good to do that before a version of Delta Chat with #7822 is published, to not increase it twice within a short time. Setting it to 1440, would increase the pixel-count by up to 25%.

Regarding the size of JPEG-metadata:

The proportion of it increases, not the amount; though, it is a relatively tiny amount of data either way. So it should not be practically relevant.

Regarding the need for more quality-testing:

I would agree, if this was about changing a reasonably good resampling-algorithm (like Triangle, CatmullRom, Lanczos3, or area-sampling) to another one, but the thumbnail-function is a performance-hack (for generating tiny preview-images), that does cause many visual flaws, and is not suitable for general image-resampling.

About performance:

I think that the < 50% time-difference is not an issue, and Triangle-resampling can always be used; though if there are reasonably common smartphones on which the difference is much larger (and practically relevant), using a resolution-condition could still be an appropriate temporary workaround.

This should be about all that i want to write about this currently. I tried to keep it short, but there is too much to consider. ^^'

I adjusted this a little, because, according to the results of the time-measurements, the difference is practically much smaller. Though, as mentioned, i think that always using triangle-resampling should be fine regarding performance.

I've measured the whole chat::send_msg() real time costs (it's what the user observes) for Triangle and for me they are up to 59.0% greater even for 6 Mpx images than ones for thumbnail(). For 4 Mpx image the worst result was +55.8%. I think it may be ok considering that 4 Mpx isn't that much. Testing on some old hardware is needed. But i see that sometimes images are 5% less in byte size, so i'd increase BALANCED_IMAGE_SIZE to 1312. Reducing traffic isn't necessary here, it's better to compensate the blur effect, even if a little.

thanks a lot for the detailed discussions!

where are the images we're optimising for here are expected to come from? 4 megapixel cameras are outdated since about 20 years - and for avatars we're using resize() already. what is the expected scope? png files and diagrams? images edited by android already?

otoh, as it does not affect most images, we can maybe be more relaxed :)

i am wondering about the only 50% slower - at #6815 (comment) we were talking about 3-4 times slower. maybe that needs to be double checked, tho also 3-4 times slower may be fine as we're not using the expensive algorithm for large images from the camera

where are the images we're optimising for here are expected to come from?

I think the PR is for screenshots mostly. At least the PR description provides screenshots.

i am wondering about the only 50% slower - at #6815 (comment) we were talking about 3-4 times slower. maybe that needs to be double checked, tho also 3-4 times slower may be fine as we're not using the expensive algorithm for large images from the camera

That time i only measured time spent on resizing, it's indeed still ~3 times slower with Triangle for me (also rechecked recently). But it's not what the user observes, the whole chat::send_msg() should be measured and it gets only ~1.5 times slower. Still, this is noticeable if i try to send photos from my 16 Mpx camera, so limiting the image size may make sense. Anyway, i'd prefer being neutral about merging this PR because there may be users having older hardware, maybe we should look for other options of improving the overall image quality first.

There are many sources for images with a resolution below 4 MP; for example:

• Zoomed in photos (at least when the camera is not capable of sufficient optical zooming)
• Screenshots of videogames/apps/videos (FullHD is 1920x1080, and QHD is 2560x1440; both are below 4 MP.)
• Digital art
• Cropped images from any source
• Photos made while a camera(-app) was set to store images at a lower resolution (useful for reducing the required storage-space).
• Images received via a Chat-app.
• The World Wide Web.

Though, ideally, a better resampling-algorithm, than the thumbnail-function, should always be used; so i prefer to figure out the highest currently acceptable¹ resolution-limit for the condition to use Triangle-resampling.

I think the PR is for screenshots mostly. At least the PR description provides screenshots.

This is intended to improve the quality of images in general, including photos, screenshots, and digital art; though the visual issues are most easily seen with text (also in photos), and other fine details, which is why i used screenshots of text as an example.

Testing on some old hardware is needed.

Indeed. Older hardware could be much slower for some reason, so it would be good to have a practical confirmation of the time-differences.
(I think that my Pixel 3a, with a Qualcomm Snapdragon 670, is probably too fast for this.)

Though, theoretically, when considering a 60% longer total processing-time² while Triangle-resampling is used, and an image was encoded with a processor that has 25% of the processing-speed of a single efficiency-core (time multiplied by 4) of a "Core i7-12700K"-CPU (in energy-saving-mode), the differences would be:

  2.0 MP:  2.7 -  1.7 =  1.0 s
  4.5 MP:  4.0 -  2.5 =  1.5 s
  8.0 MP:  5.1 -  3.2 =  1.9 s
 12.5 MP:  7.0 -  4.4 =  2.6 s
 20.0 MP:  9.9 -  6.2 =  3.7 s
 50.0 MP: 19.2 - 12.0 =  7.2 s
178.6 MP: 58.6 - 36.6 = 22.0 s

(Time between line 2608 and line 2636 in src/chat.rs; with a build without --release, to further slow the speed by 15-20%. 178.6 MP is a little below a memory-limit, that did prevent re-encoding; so could be considered an upper limit.)

Here an image of digital art, that shows a lot of aliasing, when downsampled with the Thumbnail-function:

Though, ideally, a better resampling-algorithm, than the thumbnail-function, should always be used; so i prefer to figure out the highest currently acceptable1 resolution-limit for the condition to use Triangle-resampling.

For high-resolution images thumbnail is obviously better (than for low-resolution) by its definition, and it's even said in the docs:

This method uses a fast integer algorithm where each source pixel contributes to exactly one target pixel. May give aliasing artifacts if new size is close to old size.

The reason of aliasing effects is that for lower-resolution images source pixel weights are very different (from 0.25 to 1). So probably it doesn't make much sense to use Triangle for big images, considering the performance downgrade.

Some of the quality-issues caused by the Thumbnail-function seem to become smaller as the resolution-difference increases; in particular, the grid-pattern becomes finer (which can also look worse, depending on the image).
Though, the Thumbnail-function will still cause quality-issues, even when the resolution-difference is huge, because "each source pixel contributes to exactly one target pixel".

I added images that were resampled while using the lower media_quality-setting (resampling at 11% of the original resolution), to the comparison.

Personally, i would not care much about any of the mentioned time-differences.
If i wait 12 or 19 seconds (for images with 50 MP), or, more practically relevant, 4.4 or 7 seconds (for images with 12.5 MP), does not matter much, as i would already be waiting anyway.

I tested the performance with a PinePhone¹:

The time-difference, of the send_msg-function, was 60% or less, when Triangle-resampling was used, while using a build made without --release; similar to previous measurements.

  3.6 MP:   4.3 -  2.7 =  1.6 s | 60% longer
  7.4 MP:   7.0 -  4.4 =  2.6 s | 59% longer
 12.5 MP:  11.9 -  7.8 =  4.1 s | 53% longer
 20.4 MP:  18.2 - 12.3 =  5.9 s | 48% longer
 93.0 MP:  72.2 - 51.5 = 20.7 s | 40% longer
158.0 MP: 117.2 - 84.4 = 32.8 s | 39% longer

A release-build should be about 15% faster.

I looked for some smartphones with similarly slow processors, and those seem to usually have cameras with 8-16 MP.

Resampling images with the "Image"-library is apparently as slow as it is, because it does that single-threaded; including the (de)coding of JPG-images.

158.0 MP: 117.2 - 84.4 = 32.8 s | 39% longer

Thanks for testing, but my opinion as a user is that if i sent screenshots often, i'm not sure i would be fine even with +30%, so i think we should have some linear size limit. According to your measuremts, switching to Triangle gives +2.6s in processing time for 3840x2160 on a rather slow CPU. Looks fine for me, but we need more opinions.

i would be kind of fine to give resize() for images up to 4 mp a a try. iirc, this was the initial idea i commented above.

this would not slow down processing most camera images, but already many screenshots and png would benefit. and the additional time used will be cut, as high-res images are not processed and we do not come in regions where processing takes tens of seconds longer.

i am also fine to increase the image size to 1312 px - this will also slightly improve quality, by a little bit of larger images when resize() is not used, but that's fine. we did not increase that part for years.

the PR is small enough to be reverted if that makes troubles in practise.

otherwise, in general, speed wins. while it is reasonable to optimise within speed constrains, so best quality using same speed, it is a non-goal to have the best thinkable image quality and being slow. if that is really needed, one can always send an image as file. for most user, it just does not matter.

again, thanks a lot for all these thoughts!

1344 would result in similar file-sizes in these tests.

I do not have much of an opinion about small changes to this.

I retested this and see that the most reduction in size (if we don't change BALANCED_IMAGE_SIZE) is got for:

• High-quality JPEGs, e.g. i got 420819 vs 346550 bytes for some 2400-px JPEG with quality=98.
• High-resolution JPEGs, e.g. i got 386397 vs 338597 bytes for the same JPEG resized to 4800-px (no interpolation), but with the quality reduced to 75. Ok, the difference is less noticeable, but this is just because quality=75 is rather low.
• PNGs, e.g. i got 576260 vs 488227 bytes for 1680x1050 such one.

Obviously, because such images are more detailed. You can also see that the test PNG was already exceeding the limit (500 kB) even w/o this PR. I've tried setting BALANCED_IMAGE_SIZE to 1440 to compensate the size reduction and got 619641 bytes for the same PNG which is bigger than 576260 by ~7.5%. This looks acceptable to me. 1440 is also good because it corresponds to Full HD. OTOH, for less detailed images, BALANCED_IMAGE_SIZE=1440 gives a +15--20% increase in size and i'm unsure it's an acceptable traffic increase. I'd say it makes sense to go for Full HD, but smth in the middle like BALANCED_IMAGE_SIZE=1360 and WORSE_IMAGE_SIZE=680 would be also good.

again, thanks a lot for all these thoughts!

I hope those were/are useful. Seems like a lot of information for a relatively small change. ^^

I plan to wait with changes to this pull-request, until there is some agreement for what to change it to. If some practical use is wanted before further changes, feel free to merge this pull-request as it is.

Though, i did collect some more information about increasing the resolution-limit to 1440 anyway; comparing the file-sizes, after sending 33 images¹:
The largest file-size was 833 KB (768 KB with Triangle-resampling), for a photo that has fine details everywhere in the image, and high contrast; it was about 22% larger in file-size, compared to 1280, after re-encoding.

6 images were above 500 KB, and all had a large amount of details, such as about 6-40 colourful line-art-based characters, or detailed backgrounds; all of these 6 images had rather noticeable aliasing after re-encoding, causing issues such as adding curves and rough edges to straight/curved contours, sometimes making anime-style faces look like a tongue is pushed against a cheek from the inside, or otherwise changing facial-expressions. Using Triangle-resampling, reduced the file-sizes by up to 12%, proportional to how much of an effect the aliasing caused by the Thumbnail-function had.

A generated image with a resolution of 1920x1080, that was made entirely of RGB-noise (applied to a middle-gray background, for maximising contrast), was 2.6 MB instead of 2.1 MB (~23% larger) for an image with a resolution of 1280x1280 made in the same way.
A screen-image of (japanese) text with a resolution of 3840x2160, a rainbow-gradient and 25% RGB-noise applied (lots of details, contrast, and colours, to increase file-size), was about 1.05 MB after sending.

So it seems likely, that almost all images will be below 1 MB after re-encoding, and that most images will be below 500 KB after re-encoding.

Changing the resolution-limit for the lower quality-setting, does practically change how much money (or effort, and knowledge about apps for manually reducing the file-size of images) is necessary to be able to communicate with images; so that should probably be considered in a separate pull-request.

Changing the resolution-limit for the lower quality-setting, does practically change how much money (or effort, and knowledge about apps for manually reducing the file-size of images) is necessary to be able to communicate with images; so that should probably be considered in a separate pull-request.

I have an image at hand (a quite detailed PNG) which is still smaller with this PR in "Worse quality" even if we increase WORSE_IMAGE_SIZE to 680: 147115 vs 151095 bytes w/o this PR. With WORSE_IMAGE_SIZE = 640 it compresses to 131779 bytes and some text even becomes harder to read. Basically it's the same for "Worse" and "Balanced" quality, so it makes sense to keep WORSE_IMAGE_SIZE at BALANCED_IMAGE_SIZE / 2. It's true that some images will be bigger with the limits increased to 1360 and 680 (probably, less detailed images), but the purpose of this PR isn't decreasing traffic, so i suggest to increase both limits to "compromise" values. 1440 (Full HD) may be too much indeed. Let's wait for more opinions/experiments here.

Combined file-sizes for 33 images¹ after sending:

Largest file-size of a single image among the 33 images¹ after sending: